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Exploring  the  relationship  between  distributed  training,  integrated  learning 
environments,  and  immersive  training  environments 


Introduction 

The  original  goal  of  this  paper  was  to  review  the  literature  on  the  new  areas  of 
integrated  learning  environments  and  immersive  training  environments,  as  well  as 
distance  and  team  training  and  intelligent  tutoring  systems  research,  and  evaluate  the 
feasibility  of  integrating  these  ideas. 

Background 

With  the  military's  increased  reliance  on  advanced  technology,  the  push  to 
acquire  and  maintain  proficiency  on  the  use  of  that  technology  has  increased  as  well 
(Barry  &  Runyan,  1995).  In  order  to  maintain  its  preeminence,  the  Air  Force  needs  to 
maintain  an  ability  to  use  advanced  training  technology  and  deliver  instruction  on 
demand  (Sikes,  Cherry,  Durall,  Hargrove,  &  Tingman,  1996).  New  technologies  are 
creating  novel  instructional  methods  and  techniques  as  part  of  the  design,  such  as  e- 
learning,  immersive  training  environments,  and  interactive  learning  environments. 
However,  different  views  of  how  effective  integrated  learning  environments  and 
immersive  training  environments  are,  especially  in  comparison  to  traditional 
technologically-based  training  methods,  is  not  well  known. 

While  research  is  necessary  to  ultimately  determine  effectiveness,  integration, 
and  scalability  of  the  technologies,  thoroughly  studying  what  exists  in  the  literature 
and  relating  those  findings  to  previous,  earlier  attempts  to  apply  training  technology 
can  allow  for  more  effective  focusing  of  research  efforts  and  avoid  duplication  of 
results.  The  following  short  sections  briefly  review  the  major  areas  and  provide  a  brief 
background  for  the  summaries  of  new  research  directions  discussed  in  this  paper. 


1.  Intelligent  tutoring  systems,  Distributed/Distance  Training,  Team  Training 

Intelligent  tutoring  systems.  Instruction  can  take  many  forms  such  as  classroom 
lecture,  group  activities,  and  discovery  learning;  however,  the  access  to  technology 
expands  the  possibilities  and  provides  a  potential  for  individualized  instruction.  The 
basic  outline  for  this  type  of  individualized  instruction  with  computers,  intelligent 
tutoring  systems,  has  been  around  since  1973  (Hartley  &  Sleeman,  1973);  however,  a 
real  break  through  came  when  validated  psychological  theories  about  learning  were 
applied  to  the  creation  of  these  systems  (for  summaries  see  Anderson,  Corbett, 
Koedinger,  &  Pelletier,  1995;  Corbett,  Koedinger  &  Anderson,  1997).  Although  a 
variety  of  tutoring  systems  were  proposed  and  some  developed  (Sleeman  &  Brown, 
1982;  Wenger,  1987),  those  systems  developed  by  Anderson  and  colleagues  have  been 
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used  in  public  schools  and  have  also  continued  to  inform  learning  theory  (Anderson  & 
Gluck,  2001;  Corbett  et  al.,  1997;  Koedinger,  Anderson,  Hadley  &  Mark,  1997).  These 
tutoring  systems  are  thus  unique  because  they  were  able  to  scale  up  to  a  real  world 
situation  where  many  such  systems  did  not. 

A  majority  of  individual  tutoring  systems  are  not  scalable  due  to  development 
costs,  even  though,  unlike  off-the-shelf  educational  software  (edutainment),  they 
provide  empirical  validation  of  learning.  However,  these  tutoring  systems  have 
applicability  in  distributed/ distance  training  situations,  where  trainees  are  expected  to 
do  some  or  all  of  their  learning  on  their  own  and  away  from  the  class. 

Distributed/Distance  training.  In  recent  years,  the  advancement  of  military 
technology  has  resulted  in  highly  complex  skills  that  need  to  be  acquired  and 
maintained  to  use  that  technology  (Barry  &  Ryan,  1995).  Consequently,  the  demand  for 
ongoing  innovations  including  distance  training  has  increased.  While  sending 
personnel  to  training  sites  can  be  costly,  recent  surveys  have  found  that  considerable 
cost  savings  with  comparable  training  outcomes  can  be  acquired  through  the  use  of 
distance-learning.  For  example,  the  Asynchronous  Computer  Conferencing  (ACC) 
used  by  the  Army  resulted  in  comparable  performance  between  resident  and  ACC 
students  but  cost  less  (Hahn,  1990).  However,  even  though  distance  learning  in  some 
form  has  been  available  for  many  years  (e.g.,  correspondence  courses),  studies  focusing 
on  the  internet/  web-based  training  have  only  recently  been  performed  and  relatively 
few  have  examined  the  effectiveness  of  the  technology  for  knowledge  acquisition 
(Boling,  &  Robinson,  1999;  Kerka,  1996).  With  increasing  use  of  these  technologies, 
research  focused  on  the  factors  that  could  improve  learning  from  this  type  of  training 
would  be  beneficial. 

While  research  in  intelligent  tutoring  systems  focused  on  the  individual,  issues 
in  distance  training  apply  equally  to  individuals  and  teams.  For  example,  under  ideal 
training  situations,  multiple  individuals  can  be  trained  together  to  form  effective 
teams.  However,  the  constraints  of  distance  and  time  often  play  a  role  in  limiting  team 
training.  Furthermore,  limitations  in  available  personnel  and  the  need  to  spread  these 
personnel  over  wide  geographic  areas  also  impedes  the  ability  to  train  effective  teams. 
One  solution  to  these  problems  is  distance  team  learning. 

Team/Group  training.  A  parallel  development  to  distance  training  has  occurred  in 
the  area  of  team  training.  Teams  play  an  increasingly  large  role  in  many  aspects  of 
military  work.  In  particular,  the  growing  complexity  of  tasks  frequently  surpasses  the 
cognitive  capabilities  of  individuals  and  thus,  necessitates  a  team  approach  (Cooke, 
Salas,  Cannon-Bowers,  &  Stout,  2000;  Salas,  Cannon-Bowers,  Church-Payne,  &  Smith- 
Jentsch,  1998).  However,  most  individuals  have  little  formal  training  in  how  to  work 
within  a  group,  much  less  how  to  learn  within  one.  Moreover,  the  move  toward 
distributed  mission  training  for  many  military  training  programs  creates  new 
challenges  for  collaborative  learning  by  teams.  Trainers  are  faced  with  the  problem  of 
both  training  the  skills  directly  needed  for  the  job  and  the  interpersonal  skills  needed 
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for  successful  team  performance.  In  addition,  trainers  and  trainees  must  adapt  to 
distributed  training  programs.  At  the  heart  of  both  individual  and  team  training, 
however,  is  the  ability  to  accurately  and  reliably  assess  learning  and  measure  the 
knowledge  that  results  from  training.  Although  progress  has  been  made  on 
knowledge  measurement  at  the  individual  level,  the  measurement  of  team  knowledge, 
and  team  cognition  in  general,  is  still  in  its  infancy  (Cooke  et  al.,  2000).  In  addition, 
extensive  research  has  been  performed  on  individual's  cognitive  skill  acquisition  (see 
above),  however,  team  learning  and  computer  supported  cooperative  learning  have 
only  recently  become  the  focus  of  attention,  especially  for  groups  larger  than  two 
individuals.  A  greater  understanding  of  what  an  individual  learns  within 
collaborative  learning  situations  and,  as  opposed  to  or  in  conjunction  with,  what  the 
team  learns  is  needed  in  order  to  provide  information  for  trainers. 

Distance  learning  may  have  a  different  meaning  in  the  context  of  group 
(collaborative)  or  team  training.  Few  studies  have  examined  collaborative  (group) 
distance  learning  and  even  fewer  studies  examine  team  distance  training  (Frost  & 
Fukami,  1997;  Kerka,  1996).  Computer  internet/  web  environments  have  tended  to  be 
built  for  the  individual  (Calvani,  Sorzio,  &  Varisco,  1997);  however,  groupware  and 
CMC  has  allowed  for  the  possibility  of  the  delivery  of  team  distance  learning.  Results 
from  studies  comparing  CMC  to  face-to-face  have  found  mixed  results.  For  example 
studies  have  suggested  that  CMC  can  reduce  social  norms  (Sproull  &  Kiesler,  1986), 
can  be  a  hindrance  to  the  creation  of  meaning  (Mantovani,  1996)  and  can  lead  to 
lengthy  decision  making  (Hedlund,  Ilgen,  &  Hollenbeck,  1998)  in  comparison  to  face- 
to-face  communication.  These  studies  are  limited  because  they  tend  to  focus  on 
organizational  and  social  factors  rather  than  team  process,  cognition  and  performance. 
Also,  military  teams  tend  to  be  populated  by  hierarchical  teams  of  individuals  each 
with  his  or  her  own  role,  as  opposed  to  homogeneous  groups  of  individuals  making 
the  same  judgments  or  decisions.  Therefore,  in  order  to  study  team  distance  learning, 
new  techniques  need  to  be  developed. 


2.  New  ideas  in  training:  Integrated  learning  environments  and  Immersive  training 
environments 

With  the  advent  of  more  powerful  computing,  training  systems  can  provide 
more  personalized  and  interactive  feedback  to  a  trainee.  Integrated  learning 
environments  and  immersive  training  environments  are  two  of  the  most  recent 
approaches  that  leverage  off  of  computing  power  for  improving  training. 

Immersive  training  environments  have  been  characterized  as  virtual  reality 
technology  but  also  include  a  simulation  component  (Furness,  Winn,  &  Yu,  1997). 
Simulations  and  virtual  reality  have  been  used  extensively  for  training  in  the  military 
(see  Tobias  &  Fletcher,  2000).  The  difference  between  non-immersive  versions  of  these 
technologies  and  immersive  versions  can  be  defined  in  terms  of  the  degrees  of  sensory 
impact  on  the  trainee.  Because  3D  non-immersive  environments  can  provide  very 
realistic  situations  for  the  trainee,  determining  when  to  use  the  more  expensive 


Lee 


page  8 


immersive  technologies  becomes  a  question.  Therefore,  reviewing  the  effectiveness  of 
non-immersive  and  immersive  technologies  can  be  the  first  step  before  determining  the 
relationship  between  immersive  training  environments  and  integrated  learning 
environments. 

Integrated  learning  environments  are  characterized  by  a  systems  approach  and 
provide  delivery  of  training  on  demand  (often  from  the  internet),  providing  alternative 
learning  methods,  and  easily  allow  instructor  management  (e.g.,  changes  of  materials, 
monitoring  students,  providing  correction)  (Honebein,  1996;  Nishinosono,  1992).  This 
idea  is  a  natural  extension  of  e-learning  or  web-based  instruction.  With  advancement 
in  technology  and  technological  proficiency  of  instructors,  the  suggestion  of  integrating 
various  learning  environments  not  only  makes  sense  but  also  becomes  feasible.  Thus, 
integrated  learning  environments  would  allow  an  instructor  to  provide  a  variety  of 
digital  instructional  media  (e.g.,  journal  articles,  chat  rooms,  tutoring  systems),  while 
also  allowing  instructors  to  monitor  and  maintain  contact  with  students.  However, 
many  tools  for  the  internet  are  not  integrated.  Tools  have  been  independently 
developed  and  instructors  who  would  like  to  conduct  their  classes  fully  or  partially  via 
e-learning  may  find  difficulty  in  reaching  the  goal  of  integration.  Therefore,  the  time  is 
right  for  additional  research  into  deciding  which  tools  are  critical  for  successful 
integrated  learning  environments,  the  development  of  tools  that  could  be  integrated, 
usability  issues  involving  these  tools,  and  whether  learning  is  improved  when 
instructors  adapt  these  environments.  In  addition,  because  the  shortcomings  in 
evaluations  of  large-scale  military  training  simulations  have  been  identified,  it  would 
be  useful  to  find  ways  to  tie  integrated  learning  environments  with  current  large-scale 
military  training  simulation  (Simpson,  1999a,  b). 

Immersive  training  environments  utilize  new  technology  to  provide  a  more 
sensory  approach  to  learning,  whereas  integrated  learning  environments  utilize 
combinations  of  tools  and  new  takes  on  the  learning  experience.  Relating  results  from 
the  previously  well  tested  area  of  intelligent  tutoring  systems  and  the  literatures  of 
distance  and  team  training  will  provide  additional  insights  as  to  how  these  two  areas 
can  practically  and  conceptually  be  combined  and  evaluated. 


3.  Overall  summary  of  the  technical  issues 

In  summary,  traditional  research  in  training  technology  has  focused  on  the 
individual.  These  systems  have  had  successes  and  provide  information  about 
scalability  and  empirical  evaluations  of  effectiveness  (training  validity).  In  addition, 
this  research  may  inform  distance  training,  but  will  need  degrees  of  modification  to  be 
generalized  to  team  training.  Immersive  training  environments,  on  the  other  hand, 
extend  simulation  technology  and  provide  the  trainee  with  additional  contextually- 
based  learning  experiences.  All  of  these  techniques,  tutoring  systems,  distance  and 
team  training  issues,  and  immersive  training  environments,  can  apply  ideas  derived 
from  integrated  learning  environments  and  may  do  so  to  various  extents.  Reviewing 
these  literatures  to  discover  proficiencies  and  deficiencies  in  application  will  provide 
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information  about  how  integrated  learning  environments  can  be  achieved,  as  well  as 
how  immersive  training  environments  should  be  designed  and  evaluated. 


Method 

We  focused  exclusively  on  published  empirical  studies.  We  also  included  book 
chapters  which  are  unrefereed.  Due  to  the  fact  that  there  were  so  few  published 
papers,  we  expanded  slightly  to  include  some  papers  from  very  recent  conferences. 

The  vast  majority  of  the  articles  in  the  newest  topics  of  distance  training,  team  training, 
and  virtual  learning  environments  are  opinion  pieces.  They  also  tend  to  focus  only  on 
a  single  one  of  these  topics  and  therefore  do  not  necessarily  speak  to  the  combination. 

The  other  source  for  information  is  the  internet.  On  the  internet,  you  can  often 
find  information  about  projects  on  the  topics  covered  in  this  paper  that  are  being 
performed  in  academics,  industry,  and  the  military.  Some  of  these  authors  have 
published  papers  that  are  accessible;  many  of  these  papers  are  conference  proceedings. 
However,  the  vast  majority  of  the  sites  are  just  descriptions  of  research  being 
performed.  Some  information  from  these  sources  may  be  included  where  conference 
papers  are  available  or  citations  are  at  least  made.  Although  APA  provides  for  citing 
from  the  web,  web  sources  are  often  unreliable  or  unverifiable.  So,  although  we  used 
the  internet  to  try  to  locate  additional  sources,  websites  are  not  generally  included 
below. 

Findings 

General 

Within  the  literature,  virtual  learning  environment  or  virtual  environment  for  learning 
can  be  synonymous  with  immersive  training  environment.  In  order  to  be  complete, 
though,  the  terms  virtual  reality,  simulation/ immersive  simulation,  and  training 
(learning,  tutoring  systems  or  training  systems)  should  also  be  included.  By  virtual 
environment,  though,  computer  scientists  are  generally  referring  to  something  more 
than  a  simulation.  They  are  referring  to  an  underlying  simulation  with  an  interactive 
environment  and  the  sense  for  the  user  that  he/ she  is  immersed  and  physically 
interacting.  The  tactile  (as  well  as  audio)  feedback  seems  to  be  a  critical  factor.1 


1  In  the  field  of  education,  virtual  reality  environments  can  be  broken  down  into  3 
categories:  text-based  virtual  environment,  desktop  virtual  reality,  and  immersive 
virtual  environments.  Text-based  virtual  environments  are  like  MUDs  (Multi-User 
Dungeon  or  Domain  or  Dimension)  and  MOOs  (MUD  object  oriented);  and,  desktop 
virtual  reality  is  like  simulation  or  synthetic  training  environments.  Both  of  these  lack 
the  sensory  experience  of  an  immersive  environment. 
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The  literature  consists  mostly  of  opinion  papers,  technical  papers  (e.g.,  how  to 
create  virtual  reality  or  virtual  learning  environments)  and  survey  studies  (e.g.,  what 
people  like  when  they  use  the  system. 

The  reason  these  areas  are  covered  more  than  evaluative  research  is  that  even 
though  research  has  been  conducted  in  one  form  or  another  since  the  1980s,  it  is  still  a 
relatively  young  field,  tied  to  advances  in  technology.  For  example,  to  study  teams,  it 
is  not  clear  that  all  the  necessary  equipment  for  a  true  tactile  virtual  experience  is 
available.  Head  gear  to  eye  track  is  still  cumbersome  and  it  still  requires  experts  to 
create  the  programs  to  deal  with  the  hardware  involved. 

Due  to  the  accelerated  program  schedules  required  today,  institutions  (both 
military  and  non-military)  adapt  rather  than  publish  (Whitworth,  2005).  Therefore, 
evaluative  research  is  needed  in  all  areas,  not  just  in  cutting  edge  areas  like  virtual 
learning  environments. 

However,  it  is  difficult  to  create  good  studies.  Most  labs  control  too  many 
variables  which  makes  them  less  ecologically  valid  and  hard  to  transfer  outside  of  the 
lab.  If  only  a  few  variables  are  controlled,  then  it  is  hard  to  publish  because  traditional 
psychologists  (or  even  cognitive  scientists)  may  view  the  study  as  too  uncontrolled.  If 
a  study  is  performed  in  the  field,  then  so  many  variables  are  uncontrolled  that  often  no 
real  conclusions  can  be  drawn. 


What  do  we  know? 

Simulations 

Computer  simulations  are  used  in  just  about  every  field  to  re-create  some 
phenomenon;  however,  since  it  is  based  upon  the  programmer's  conceptualization  of 
the  phenomenon  and  the  technology  available,  the  better  the  technology  and 
conceptualization,  the  better  the  simulation.  Although  simulations  can  be  used  for 
testing  equipment,  a  primary  use  has  been  for  educating  and  training  (O'Keefe  IV  & 
McIntyre  III,  2006).  Topics  vary  widely  from  standard  educational  topics  such  as 
teaching  kids  physics,  mechanics,  and  weather  (Gould,  1995;  Mayer,  2003),  to  practical 
topics  such  as  flight  simulation  (Gopher,  Weil,  &  Baraket,  1994),  nursing  (Hamilton, 
2005),  business  (Marriott,  2004;  Valenzuela,  2003),  biomechanics  (Schwameder  & 
Muller,  2001),  chemistry  structures  (Danielsson,  Banushkina,  Nutt  &  Meuwly,  2006), 
and  urban  low  intensity  conflict  environments  (LIC;  Kiperman,  E.,  Sirkis,  A.,  & 
Popovich,  2005).  Virtual  learning  environments  are  dependent  upon  the  "goodness"  of 
the  underlying  simulation  -  the  people  doing  the  conceptualization  of  the 
phenomenon  such  as  psychologists,  scientists,  mathematicians,  and  programmers,  the 
technology,  and  the  programmer's  abilities. 
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Teams 

Group  learning  and  collaboration  have  been  studied  in  social  psychology  and 
education.  The  connection  between  this  field  and  technology  started  with  computer 
supported  cooperative  work  (CSCW;  Olson  &  Olson,  1991).  The  idea  in  CSCW, 
though,  was  not  for  learning  but  improving  people's  ability  to  work  together. 
Computer  supported  cooperative  learning  (CSCL)  grew  out  of  CSCW  (Resta,  1995)  but 
is  mainly  focused  in  an  educational  setting.  Groupware  is  part  of  CSCW  (Ellis,  Gibbs 
&  Rein,  1991;  Kline  &  McGrath,  1999).  Although  originally  designed  to  aid  groups, 
and  clearly  could  be  extended  to  cover  CSCL,  research  to  validate  has  not  been 
extensive  (see  Kline  &  McGrath,  1999.)  However,  researchers  in  the  recent  team 
literature  prefer  the  term  team  over  group  because  a  team  has  a  shared  mission 
whereas  a  group  may  not  have  any  reason  to  be  together  (Paris,  Salas,  &  Cannon- 
Bowers,  2000).  (Note  that  as  in  much  of  the  literature,  everything  is 
compartmentalized.  People,  researchers  and  others,  rarely  talk  across  the  lines  of  the 
groups  that  have  been  formed.  There  may  be  relevant  research  within  groupware  but 
that's  somewhat  beyond  the  scope  of  this  paper.)  The  rise  in  team  research  in  the 
cognitive  science  area  has  paralleled  a  decrease  in  group  studies  in  social  psychology 
(Druckman,  2004;  researchers  such  as  Mooreland  are  still  studying  these  issues, 
though.) 

Although  team  research  could  be  linked  to  collaboration  research  (Levine  & 
Moorleand,  1990),  recent  research  has  emerged  from  the  intersection  of  cognitive 
science  and  human-computer  interaction  (for  review  see  Paris  et  al.,  2000;  Salas, 

Bowers  &  Cannon-Bowers,  1995).  The  research  has  focused  on  improving  methods  for 
measuring  and  improving  team  performance  (Cooke,  2005;  Cooke  et  al.,  2000;  Driskoll 
&  Salas,  1991;  Dwyer,  Oser,  Salas,  &  Lowlkes,  1999;  Winfred,  A.,  Edwards,  B.,  Bell,  S.T., 
Villado,  A.,  &  Bennett,  W.,  2005;  Wright  &  Kaber,  2005).  Various  factors  can  affect 
team  performance  such  as  the  usability  of  the  system  (i.e.,  interface,  how  intimidating, 
how  realistic,  etc.),  how  evenly  the  work  is  distributed,  whether  the  individuals  have 
high  cognitive  ability,  and  the  structure  of  the  team  itself  (Ellis,  Hollenbeck,  Ilgen, 
Porter,  &  West,  2003;  Lee,  Bond,  Scarbrough,  Gillan,  &  Cooke,  2006;  see  also,  Salas, 
Cannon-bowers,  &  Johnston,  1997;  Stata,  1989).  Many  ways  have  been  found  to 
improve  team  performance  (coordination  training,  leadership  training,  self-correction, 
for  review  see  Cannon-Bowers  &  Salas,  1998).  One  example  is  cross-training  (Cannon- 
Bowers,  Salas,  Blickensderfer,  &  Bowers,  1998;  McCann,  Baranski,  Thompson  & 

Pigeau,  2000;  Volpe,  Cannon-Bowers,  Salas,  &  Spector,  1995).  However,  as  with  virtual 
environments,  systematic  evaluation  of  systems  needs  to  be  performed  more  often 
(Salas,  Burke,  Bowers,  &  Wilson,  2001). 

Cognitive  variables  must  be  combined  with  the  social  variables.  Because  team 
learning  is  also  a  social  endeavor,  knowledge  gained  by  one  member  can  pass  to  other 
members  (Gruenfeld,  Martorana,  &  Fan,  2000;  Strasser,  1992).  Thus,  not  only  does 
training  in  skills  such  as  how  to  communicate  with  other  team  members  help  but  also 
the  analysis  would  be  incomplete  without  examining  cultural  cognition  variables  (such 
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as  individualism/ collectivism  or  agreeableness  (Lee  et  al.,  2006).  Team  training  does 
appear  to  work  but  factors  outside  of  improved  technology  may  be  involved. 

Simulations  have  been  used  not  just  for  team  training  but  also  for  merely 
measuring  team  behaviors;  however,  virtual  reality  environments  are  still  quite  new. 
For  example,  students  can  learn  how  to  fly  an  unmanned  air  vehicle  (Cooke  &  Shope, 
2004).  The  US  DoD  has  funded  the  Phoenix  project  which  allows  teams  of  engineers  to 
design  together  using  a  simulation  (Ng,  Guleyupoglu,  Segaria,  Malone,  Woyak,  & 
Salow,  2003;  see  also  website  such  as  Malone,  2006  and  Lee  et  al.,  2006)  where  teams 
can  use  a  peacekeeping  simulation  of  Bosnia  (software  by  Aptima  Corporation).  As 
good  as  these  simulations  are,  however,  they  do  not  approach  actual  virtual  learning 
environments  where  the  team  would  be  completely  immersed  in  the  technology  (see 
further  discussion  below).  Work  in  this  area  is  proceeding  but  not  yet  appearing  in  the 
published  journal  literature.  (For  one  recent  study  combining  teams  and  virtual  reality 
but  not  distance,  see  virtual  reality  section.) 


Distance 

The  military  has  made  a  commitment  to  distance  learning  with  the  latest  computer 
technology  (e-learning).  In  a  review  of  the  factors  affecting  distance  on-line  learning, 
Thurston  &  Earnhardt  (2003)  note  that  completion  rates  for  such  courses  is  not  as  high 
as  traditional  courses  and  the  factors  affecting  completion  may  include  outside 
distractions,  distractions  in  the  course  itself  (hardware/ software  problems),  personal 
commitment  to  the  course  (goals),  and  timely  feedback  on  performance.  The  difference 
between  personal  learning  and  required  learning  is  that  the  individual  in  a  personal 
learning  situation  may  not  feel  obligated  to  complete.  However,  if  a  team  of 
individuals  is  being  trained  together,  social  pressure  may  create  another  level  of 
obligations  that  helps  individuals  to  complete  training.  In  some  cases,  team  distance 
training  can  be  as  good  as  or  even  better  than  team  co-located  training  (Lee  et  al.,  2006). 
Essentially,  co-located  training  and  testing  may  cause  distractions,  especially  with 
highly  complex  technology  or  environments.  The  problem  is  that  one  cannot  just  take 
existing  software  and  expect  it  to  work  in  a  distributed/ distance  fashion  -  either  for 
individuals  or  for  teams  (MacDonald,  2005).  Two  recent  studies  are  described  next. 

1.  In  a  study  performed  by  Beranek  &  Martz  (2005),  teams  of  students  were  asked  to 
work  together  over  the  internet  in  order  to  improve  communication.  (No  learning 
evaluations  were  performed.)  Teams  receiving  training  on  how  to  interact  effectively 
performed  better  on  qualitative  measures.  This  study  shows  how  confusing  the 
nomenclature  is.  In  this  sense,  the  virtual  team  was  virtual  because  they  were 
separated  by  a  distance  and  the  environment  was  virtual  because  it  wasn't  in  one 
physical  location;  however,  it  was  not  a  virtual  learning  environment  in  the  sense  that 
team  members  did  not  perform  tasks  together  represented  in  a  fully  immersive, 
simulated  environment. 
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2.  In  a  study  by  Dornheim  (2000),  a  distributed  mission  was  arranged  between  8  pilots 
using  a  high-fidelity  F-15C  simulators.  Although  performance  was  not  evaluated, 
pilots  comments  were  studied.  The  issues  that  arose  were  that  simulators  need  to  be 
more  realistic,  and  developers  need  to  reduce  the  system  delay.  In  addition,  budget 
restrictions  prevent  the  development  of  full-scale  implementations. 


Virtual  reality /Virtual  environments 

Virtual  environments  and  interactivity  have  been  the  focus  of  attention  for 
several  decades  now  (Conn,  Lanier,  Minsky,  Fisher,  &  Druin,  1989;  Macredie,  Taylor, 
Yu  &  Keeble,  1996).  While  the  focus  by  developers  will  always  be  a  bit  skewed  toward 
technology,  recent  developments  have  also  considered  the  importance  of  human 
interaction  (Card,  Moran  &  Newell,  1998;  see  also  Middleton,  McIntyre  III,  &  O'Keefe 
IV,  1993).  This  is  fortuitous  because  virtual  reality  requires  more  than  a  faithful 
simulation  but  also  an  ability  for  a  human  to  be  part  of  the  whole  system. 

Research:  Durlach  et  al.  (2000)  presents  a  review  of  virtual  environments; 
however,  the  citations  mainly  fall  in  conference  proceedings  and  technical  reports. 
While  this  would  be  a  good  initial  read  for  an  introduction  to  the  topic,  Stanney's  2002 
book  provides  quite  a  bit  more  solid  empirical  data  within  the  chapters. 

Although  more  papers  focus  on  commentary  or  technical  descriptions,  some 
papers  did  provide  empirical  evaluation  of  one  type  or  another.  This  section 
summarizes  relevant  papers  by  topic. 

1.  Virtual  humans: 

People  will  respond  to  virtual  humans  in  a  more  realistic  way  when  those  virtual 
humans  respond  to  them  more  realistically  (Garau,  Slater,  Pertaub  &  Razzaque,  2005). 
The  sense  is  that  the  technology  still  has  a  ways  to  go  before  people  will  respond  to 
virtual  humans  in  the  same  way  but  this  may  change  as  people  have  more  experience 
with  immersive  environments. 

2.  Fine  motor  skills: 

Arnold,  Farrell,  Pettifer,  &  West  (2002);  The  beginning  of  this  paper  reviews  research 
on  gross  motor  skills,  with  the  general  findings  that  people  can  learn  but  do  not 
necessarily  perform  better  than  with  real  world  tasks.  The  task  involved  movement  of 
an  object  through  a  physical  space,  as  opposed  to  merely  picking  up  and  grasping.  In 
three  experiments,  the  VE  was  more  difficult  than  the  real  world  task.  In  the 
discussion,  the  authors  compare  their  work  with  previous  studies,  notably  Kozak, 
Hancock,  Arthur  &  Chrysler  (1993)  who  have  found  less  difficulty  with  VE.  The  main 
finding  is  that  VE  is  different  from  the  real  world  and  therefore  in  order  to  fully  test  a 
system  different  tasks  of  varying  types  must  be  used  to  evaluate. 
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3.  Building  a  database  of  results: 

Loughran  (1998)  presents  an  idea  and  the  first  steps  toward  creating  a  virtual  training 
repository.  They  collected  about  12  months  of  data  from  the  army's  program  and 
found  that  was  difficult.  The  process  of  archiving  was  time  consuming,  error  prone, 
and  required  constant  oversight.  Thus,  although  this  seems  like  a  fantastic  idea, 
especially  so  that  future  researchers  can  analyze  performance  data,  the  system  still  has 
a  long  way  to  go. 

4.  Evaluation  instruments: 

The  easiest  and  most  often  used  measure  of  learning  in  all  tutoring  and  training 
systems  is  the  questionnaire.  The  most  basic  asks  participants  how  they  view  the 
system  (how  effective  is  it,  do  you  use  it,  and  the  like).  For  example,  Ellaway, 
Dewhurst  &  McLeod  (2004)  present  a  longitudinal  study  of  a  very  detailed  instrument 
presented  to  medical  students.  This  type  of  instrument  is  useful  but  does  not  really 
measure  learning.  A  second  type  of  questionnaire  asks  participants  to  answer 
questions  directly  on  both  the  facts  and  ideas  presented  or  to  describe  the  actual 
procedures  learned.  This  type  of  declarative  test  is  getting  closer  to  evaluating  what 
the  learner  has  obtained  from  the  content  of  the  training  system.  A  third  type  of  test 
involves  asking  the  person  to  perform  the  procedures  taught.  These  last  two  types 
(employed  extensively  by  developers  of  intelligent  tutoring  systems)  target  the  actual 
knowledge  learned  and  carry  much  more  value  than  mere  questionnaires.  Taken 
together,  using  all  three  types  of  instruments  is  the  most  ideal  situation.  A  system  that 
is  used  and  appealing  and  effective  in  training  the  target  knowledge  is  best. 

In  Dobson  et  al.  (2001),  the  researchers  describe  a  system  where  individuals  can 
train  with  simulated  team  members.  The  system  includes  a  way  to  capture  who  is 
interacting  with  whom  and  thereby  show  whether  the  learner  is  interacting  with  the 
correct  individuals  so  that  they  will  eventually  perform  well  as  a  team.  Thus,  tools  to 
improve  analysis  of  user  actions  (or  communications)  may  be  useful,  as  well.2 

5.  Problems  with  technology: 

a)  A  longitudinal  evaluation  of  a  virtual  environment  conducted  by  Tromp,  Steed  & 
Wilson  (2003)  found  that  the  network  technology  was  not  good  enough  to  provide 
continuous  feed  and  that  generally  people  found  the  limitations  of  the  technology 
caused  frustration  (p.257).  This  study  did  not  examine  training  and  learning,  and 
therefore  illustrates  how  far  technology  needs  to  advance  before  training  and  learning 
issues  can  become  the  primary  focus. 


2  Recent  research  on  automatic  team  communication  has  shown  promise  for  Latent 
Semantic  Analysis  (Foltz,  Laham,  &  Derr,  2003;  Foltz  &  Martin,  2004;  Foltz,  Martin, 
Abdelali,  Rosenstein  &  Oberbreckling,  2006).  The  system  has  done  well  in  predicting 
overall  team  performance  through  analysis  of  the  (oral)  team  communication  collected. 
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b)  Linebarger,  Janneck  &  Kessler  (2004).  This  study  focused  on  groups  using  virtual 
reality  (not  immersed)  to  perform  a  design  task.  Essentially  issues  with  software  and 
hardware  were  the  major  obstacles  in  task  performance. 

c)  Park  &  Kenyon  (1999):  This  study  measured  human  performance  when  latency  and 
jitter  in  networks  was  varied.  Previous  exposure  to  the  task  helped  but  performance 
was  ultimately  affected  by  network  conditions.  Long  latencies  and  large  amounts  of 
jitter  had  their  greatest  impact  on  performance  during  difficult  hand  guidance  tasks. 

6.  Transfer3: 

a)  Regian  (1997)  performed  an  experiment  that  consisted  of  two  conditions:  virtual 
environment  and  a  simulation.  The  virtual  reality  environment  was  not  immersive. 
Results  indicated  that  participants  in  both  conditions  were  able  to  transfer  to  real 
world  tasks;  however,  because  the  system  was  not  complex,  it  is  hard  to  know  whether 
these  results  would  scale. 

b)  Witmer,  Bailey  &  Knerr  (1996):  A  virtual  environment  for  a  building  was  compared 
with  verbal  rehearsal  with  a  computer  model.  Transfer  to  an  actual  building  was 
better  for  those  who  trained  in  a  virtual  environment  (faster  time). 

c)  Tate,  Sibert,  &  King  (1997):  A  virtual  environment  training  for  firefighters  was 
compared  with  non-virtual  environment  training;  slight  advantages  during  test 
(actually  locating  fires  in  actual  shipboard  locations)  were  found  for  participants  in  VR 
training. 

d)  Seymour,  Gallagher,  Roman,  O'Brien,  &  Bansai  (2002):  [Note  that  other  studies  in 
medicine  are  discussed  in  the  Medicine  and  Education  section.]  Sixteen  residents  were 
randomly  assigned  to  a  virtual  reality  training  or  standard  training  group.  There  were 
no  initial  differences  in  beginning  knowledge  but  virtual  reality  training  participants 
were  faster  at  performing  procedures  and  made  fewer  errors. 

7.  Therapy: 

a)  North,  North  &  Coble  (2002):  This  research  summarizes  case  studies  used  to 
document  therapists'  use  of  virtual  reality  to  treat  patients.  Virtual  environments 
provide  patients  with  a  way  to  experience  events  in  a  simulated  world.  Because  the 
environment  is  realistic  and  the  individuals  are  immersed  in  those  environments, 
virtual  environments  provide  safe  places  to  work  on  various  mental  illnesses.  Note 
that  these  are  case  studies  and  not  empirical  data. 


3  This  report  is  focused  on  published  studies  because  they  provide  more  back 
validation  than  studies  presented  at  conferences.  One  transfer  study  not  included: 

Hall,  Stiles,  &  Horwitz  (1998)  presented  a  study  comparing  individuals  learning  in  an 
immersed  virtual  reality  versus  a  2D  environment.  The  VR  group  showed  initial 
benefits  but  after  2  weeks  both  groups'  performance  was  about  the  same.  In  addition, 
the  VR  group  experienced  nausea  and  other  physical  symptoms  during  initial  learning. 
No  followup  or  published  studies  were  found  by  us  (but  could  exist).  Therefore  it 
wasn't  included  officially  in  the  list. 
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b)  See  section  following  on  Medicine  &  Education;  Satava  &  Jones  (2002)  also  provide 
a  review  of  medical  applications  in  virtual  environments. 

8.  General  evidence  of  the  effectiveness  of  virtual  environments  for  learning  has  been 
found  by  several  researchers: 

a)  Regian  &  Shebilske  (1992):  They  developed  a  virtual  reality  maze.  Participants 
were  able  to  be  trained  very  quickly  and  perform  the  tests  very  quickly. 

Unfortunately,  the  authors  did  not  test  to  see  whether  virtual  training  would  transfer 
to  actual  maze  use. 

b)  Slater,  Sadagic,  Usoh,  &  Schroeder  (2000):  These  researchers  ran  teams  of  3 
individuals  in  a  virtual  reality  to  measure  group  interactions.  The  task  was  solving  a 
puzzle.  They  did  not  measure  either  learning  or  performance  improvement.  Only  one 
participant  could  see  his  own  figure  in  the  space.  (This  seems  to  be  a  drawback  of  the 
virtual  environment.  Ideally,  we'd  want  to  be  examining  a  virtual  environment  for 
team  training  where  all  team  members  are  not  only  represented  but  also  immersed.) 

c)  Swan,  Gabbard,  Hix,  Schulman  &  Kim  (2003):  The  researchers  compared  different 
interface  versions  of  the  same  underlying  virtual  environment.  Differences  between 
the  four  interfaces  were  found.  These  results  imply  that  the  user-interface  to  a  virtual 
environment  is  also  an  important  factor  for  performance. 

d)  Witmer  &  Sadowski  (1998):  Participants  were  asked  to  use  a  virtual  reality  system 
or  walk  in  real  life  and  with  either  feedback  or  no  feedback.  Results  indicated  that 
distance  judgments  after  using  the  virtual  reality  system  was  short  compared  to  real 
life  training. 

e)  Heldal,  Steed,  Spante,  &  Schroeder  (2005):  An  exploratory  study  was  performed 
where  6  pairs  of  participants  were  asked  to  use  a  virtual  environment.  Each  member 
of  a  pair  was  in  a  different  location.  They  were  just  asked  to  interact;  they  were  not 
asked  to  learn  anything  in  particular  or  any  task  in  particular.  The  researchers  were 
merely  observing  behaviors. 

9.  Leadership  Steed,  Slater,  Sadagic,  Bullock  &  Tromp  (1999):  This  study  investigated 
how  a  small  group  of  three  strangers  would  interact  when  brought  together  in  a  virtual 
environment  to  perform  a  task.  In  the  first  experiment,  ten  teams  were  studied;  but, 
only  one  person  was  immersed  in  the  environment.  Results  suggested  that  the 
immersed  person  tended  to  emerge  as  a  leader  in  the  virtual  group,  but  not  in  the  real 
meeting.  In  the  second  experiment,  4  groups  of  three  participants  were  studied  and  no 
participants  were  immersed.  No  special  patterns  of  results  were  found.  Thus,  the 
question  of  how  virtual  learning  environments  promote  leadership  is  still  open. 

10.  Negative  side  effects:a)  Harm  (2002):  The  first  part  of  the  article  reviews  physical 
problems  such  as  motion  sickness  and  lists  many  citations  for  previous  research.  As 
many  as  30%  of  VR  users  experience  such  severe  motion  sickness  symptoms  that  they 
stop  use.  As  many  as  95%  of  VR  users  experience  symptoms  severe  enough  to 
interfere  with  training  in  the  VR  system.  One  can  administer  medication  to  overcome 
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side  effects.  Other  methods  include  biofeedback,  acupressure,  acustimulation,  and 
adaptation  techniques. 

Cobb,  Nichols,  Ramey,  &  Wilson  (1999):  Nine  individual  experiments  were 
performed.  Demonstration  of  sickness  was  found  in  all  studies  and  across  all  measures. 
Although  symptoms  were  often  mild  and  short  lived,  in  some  cases,  the  symptoms 
were  so  severe  that  participants  had  to  withdraw. 

b)  Lawson,  Graeber,  Mead  &  Muth  (2002).  This  is  a  second  chapter  in  the  same 
handbook  that  deals  with  symptoms  of  virtual  reality  use.  Adverse  signs  and 
symptoms  are  experienced  by  at  least  10%  and  sometimes  as  many  as  90%  of  simulator 
trainees.  The  effects  include  eye  strain,  fatigue,  drowsiness,  sweating,  headache,  and 
difficulty  concentrating.  Approximately  40%  of  simulator  pilots  have  reported  no 
symptoms,  and  25%  have  reported  only  mild  symptoms.  However,  about  5%  of 
participants  had  such  severe  symptoms  that  they  had  to  stop  testing  until  the 
symptoms  subsided.  Testing  performed  on  civilians  showed  that  at  least  60%  of  VE 
users  will  report  adverse  symptoms  during  their  first  exposure. 

c)  Viirre  &  Bush  (2002):  This  is  the  third  chapter  in  the  same  handbook  that  deals  with 
symptoms  of  virtual  reality  use.  The  symptoms  identified  by  the  authors  are  eye 
strain,  seizures,  migraines,  noise  levels,  skin  irritation,  and  physical  injury. 

d)  Regan  (1997):  A  high  incidence  of  self-report  symptoms  (such  as  nausea)  was 
measured  (61%)  and  possible  causes  discussed. 

e)  DiZio  &  Lockner  (1992;  1997a,b):  This  paper  presents  a  research  program  that  not 
only  examined  motion  sickness  associated  with  use  of  head  mounted  displays  in 
virtual  environments  but  also  ways  to  improve  virtual  environments  to  avoid  sickness. 
The  head  mounted  display  itself  does  not  cause  the  problem,  symptoms  can  start  with 
an  exposure  of  only  15  minutes  but  in  some  cases,  it  will  also  decay  within  15  minutes 
after  the  exposure.  Suggestions  include  improving  visual  delay  between  head  motion 
and  system  response  and  asking  subject  to  limit  head  movements. 

DiZio  &  Lockner  (2002):  This  book  chapter  is  a  review  of  their  other  work  on 
the  physical  effects  of  virtual  environment  use.  The  authors  point  out  that  using  an 
immersed  virtual  environment  may  have  physical  effects  in  the  initial  adaptation  and 
then  subsequent  return  to  reality. 

f)  Magee  (1997):  A  virtual  reality  simulator  of  a  ship  was  created.  Twenty-six  students 
participated  in  a  longitudinal  study.  Over  4  sessions,  all  students  demonstrated 
increased  learning  and  ability  to  transfer  their  training  to  a  real  shipboard  situation. 
However,  no  control  group  was  used  and  many  suffered  simulator-induced  illness. 

g)  Stoffregen,  Draper,  Kennedy,  &  Compton  (2002):  This  chapter  gives  a  good 
overview  of  research  focused  on  vestibular  adaptation.  Incorrectly  calibrated  head 
trackers  can  influence  vestibulo-occular  reflex  reactions. 


Virtual  environments  for  individuals  seem  better  developed  than  those  for 
teams.  However,  this  is  not  surprising  because  the  technology  needed  in  order  to 
create  an  environment  where  multiple  players  can  interact  is  still  developing. 
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Although  simulations  such  as  those  described  in  the  team  section  are  becoming  more 
advanced,  the  physical  aspects  (particularly  tactile)  are  difficult  to  simulate  for 
multiple  individuals  concurrently  and  in  real  time.  To  have  one  individual  interacting 
with  simulated  team  members  seems  easier  than  having  many  real  team  members 
interacting  within  a  virtual  environment  in  real  time.  (Note  that  an  individual 
interacting  with  simulated  team  members  can  still  have  a  huge  benefit  -  both  for  the 
individual  and  for  the  people  who  want  to  train  that  individual.  It  is  often  extremely 
difficult  to  assemble  all  the  relevant  people  needed  for  team  training.)  Technology  and 
research  on  team  virtual  learning  environments  will  need  to  address  the  large  quantity 
of  sensory  data  (both  from  the  environment  and  from  all  the  team  members  actions 
and  interactions)  (Fisher,  2002). 4 

Medicine  and  Education 

Both  medicine  and  education  have  focused  much  attention  on  integrating  new 
technology  to  improve  training.  In  addition,  medicine  has  many  common  features  to 
military  training.  For  example,  training  in  medicine  can  be  high  stakes,  with  the  safety 
of  the  patient  paramount.  Further,  medicine  requires  both  adherence  to  a  hierarchy  of 
command  but  also  teamwork  at  various  levels.  With  high  costs,  many  rural  areas 
cannot  afford  full  medical  facilities  and  therefore  distance  training  and  even  distance 
procedures  are  being  examined.  Although  summarized  briefly  in  this  section,  the 
developments  in  these  areas  over  the  next  few  years,  combined  with  developments  in 
pure  technology  areas,  could  inform  the  military  in  their  own  advancements  in  virtual 
learning  environments. 

Medicine  has  used  real  actors  to  simulate  patients  (Adams,  2003;  Barrows,  1985, 
1993;  Harden  &  Collins,  1998;  Ker  et  al.,  2005)  but  computer  simulations  can  not  only 
create  adequate  substitutions  but  also  repeatable  clinical  experiences  (Objective 
Structural  Clinical  Exams  -  OSCE  -  Bradley  &  Humphris,  1999;  Kaufman  et  al.,  2000; 
Medina,  Racadio,  &  Schwid,  2000;  Shaffer,  Gordon  &  Bennett,  2004).  Thus,  the  use  of 
fully  simulated  (computer)  patients  has  become  more  routine  (for  survey  see  Liu, 
Tendick,  Cleary  &  Kaufmann,  2003).  For  example,  Hemman  (2005)  evaluated  a 
simulation  of  a  virtual  patient  in  combat  medic  training.  Three  classes  used  the  virtual 
patient  for  training  and  the  other  three  classes  had  traditional  training.  Although  no 
differences  were  found,  this  study  is  fairly  unique  because  it  summarizes  an  actual 


4  The  literature  on  teamwork  has  more  studies  using  simulations  or  synthetic  task 
environments.  Virtual  learning  environments  where  teams  are  immersed  in  virtual 
reality  are  hard  to  find.  One  exception  is  Brna  &  Aspin  (1998)  but  this  study  is 
extremely  exploratory  with  a  very  small  group  of  dyads  and  no  real  learning 
evaluation.  Another  example  is  Hudson,  Helser,  Sonnenwald  &  Whitton  (2003)  in 
which  a  collaborative  virtual  environment,  not  immersive,  was  used  collaboratively 
and  at  a  distance  by  groups  of  students.  The  system  was  a  collaborative  science  work 
experience  but  not  a  training  system.  However,  it  does  demonstrate  that  such  a  system 
can  be  created  (although  the  immersive  nature  is  still  not  validated). 
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evaluation  and  may  indicate  that  the  use  of  simulators  can  aid  in  the  training  of  future 
medics  with  detriments. 

The  advancement  of  technology  has  also  allowed  for  surgeons  to  perform 
surgeries  at  a  distance  (Eadie,  Seifalian,  Davidson,  2003;  Jones,  1997).  Although  most 
studies  report  success  (Eadie  et  al.,  1997),  improvements  in  technology  and  training 
students  on  how  to  use  such  technology  are  needed  to  advance  the  field.  In  addition, 
evidence  that  virtual  reality  training  can  improve  surgical  performance  can  be  found  in 
several  publications  (see  Cosman,  Cregan,  Martin,  &  Cartmill,  2002;  Harir,  Srivastava, 
Youngblood,  &  Ladd,  2004;  Seymour  et  al.,  2002)  and  can  in  some  cases  transfer  that 
training  to  a  real  life  situation  (Seymour  et  al.,  2002);  but  more  work  needs  to  be  done 
to  extensively  compare  current  methods  with  virtual  methods  (Cosman  et  al.,  2002). 

The  inclusion  of  haptic  devices,  has  made  surgical  training  more  realistic  and  effective 
(Liou  et  al.,  2003);  however,  these  are  still  not  immersive,  with  simulations  of  surgical 
team  members.  (Note  that  virtual  reality  can  also  be  used  to  train  patients.  For 
example,  Pons  et  al.  (2005)  used  virtual  reality  to  train  patients  who  had  lost  a  limb  to 
use  a  hand  prosthesis.  Other  studies  are  described  in  Holden  &  Todorov  (2002).) 

Teamwork  is  also  becoming  a  focus  in  medical  education  (Clark,  Leinhaas,  & 
Filinson,  2002).  Although  virtual  reality/ virtual  learning  environments  do  exist  in 
medical  education  (see  for  a  review,  Satava  &  Jones,  2002),  the  combination  with 
teamwork  and  distributed  training  is  hard  to  find  in  printed  journal  articles. 

Evaluation  of  applications  of  virtual  reality  to  both  group/ team  training  and  medicine 
are  starting  to  appear  in  the  conference  literature  and  should  start  appearing  in 
publications  soon  (see  Shaffer  et  al.,  2004  for  ideas  about  evaluation). 

In  education,  team  learning  has  often  been  referred  to  as  collaborative  learning 
or  small  group  learning  (for  one  review  see  Johnson,  Johnson  &  Smith,  1991;  Springer, 
Stanne  &  Donovan,  1999).  The  ideas  have  moved  from  the  classroom  to  the  web. 
Although  web-based  distance  learning  has  recently  become  popular,  the  vast  majority 
of  the  students  still  attend  classes  at  a  campus  and  multimedia,  as  well  as  web-based 
distance  learning,  is  used  more  as  a  supplement  (Schulz  &  Dahale,  1999).  Successful 
training  using  multimedia  learning  is  well-documented  (see  work  by  Mayer  for 
multimedia  instruction.)  However,  in  general,  evaluations  in  web-based  learning  and 
applications  found  in  schools  or  other  educational  venues,  tend  to  focus  on  satisfaction 
more  than  on  learning  outcomes  and  therefore,  as  with  all  areas,  more  evaluative 
research  is  needed.5  On  the  other  hand,  although  simulations  and  web/ distance 
learning  have  been  adapted  to  a  certain  extent  (Anderson  &  Jackson,  2000),  virtual 
environments  have  not  been  so  easily  integrated.  To  start,  virtual  learning 
environments  can  mean  many  things  to  many  authors /researchers.  For  example, 
Akeroyd  (2005),  defines  many  systems  used  in  the  UK  as  virtual  learning 


5  Distance  learning  has  been  used  in  education  for  quite  a  long  time.  Navarro  & 
Shoemaker  (2000)  argue  that  most  evaluations  and  opinions  are  still  based  upon  the 
oldest  technologies.  Their  studies  show  that  web-based  learning  can  be  effective, 
however,  again,  they  have  not  evaluated  immersive  virtual  environments. 


Lee 


page  20 


environments,  but  does  not  really  mean  the  use  of  virtual  reality,  so  much  as  that  the 
participants  are  virtual  (i.e.,  not  present).  He  discusses  the  benefits  and  authoring 
issues  involved  with  essentially  web-based  distance  learning;  however,  virtual  reality 
is  not  really  included.6  Thus,  some  common  terminology  needs  to  be  imposed  in  order 
to  be  able  to  equate  work  across  disciplines. 

Another  issue  is  that  the  technology  needed  for  students  at  a  distance  to  have  a 
true  virtual  experience  is  too  expensive  to  be  widely  available.  Further,  groupware  has 
not  always  been  the  focus  of  attention  (Anderson  &  Jackson,  2000).  Whitworth  (2005) 
argues  that  either  instructors  are  not  adopting  or  they  are  adopting  and  not 
publicizing.  In  either  case,  evaluative  research  for  virtual  learning  environments  has 
not  appeared  as  readily  in  the  literature.7  It  is  likely  that  virtual  environments  on 
campus  will  appear  before  distance  virtual  environments  but  both  developments  will 
depend  upon  advancements  in  technology. 

From  a  quick  read  of  these  areas,  medicine  and  education,  it  is  clear  that 
advances  in  virtual  learning  environments  and  evaluations  of  systems  developed  in 
these  areas,  are  likely  to  be  very  promising  and  any  future  researchers  should  pay 
attention  to  innovations  in  these  areas. 


Recommendations  for  promising  areas  of  research  with  a  discussion  of  the  problems 
and  scalability  of  these  approaches  for  current  Air  Force  training  practices. 


Potential  advantages 

As  technology  improves  and  becomes  less  expensive,  it  may  become  more 
viable  to  have  immersive  virtual  environments  at  a  distance  and  with  teams  and  the 
individuals'  experiences  may  be  no  different  from  when  they  are  training  individually 
in  one  location. 

•  VLE  can  provide  a  more  realistic  environment.  They  allow  for  the  individual 
and  team  members  to  have  more  repetitions  on  task,  (see  Shaffer  et  al.,  2004) 

•  Improved  distance  and  team  VLE  could  allow  for  an  expert  (or  instructor)  to 
be  available  to  interact  virtually.  (More  than  one  expert/  instructor  might  also  be 
available). 


6  Other  researchers  have  also  used  virtual  learning  environment  merely  to  mean 
students  (and  teachers)  working  on  the  internet  rather  than  a  classroom  (see  Weller, 
Pegler  &  Mason,  2005,  for  another  example).  Thus,  definition  is  a  continual  problem 
when  researching  these  issues. 

7  In  a  pilot  study,  Johnson,  et  al.  (1998)  evaluated  an  immersive  virtual  environment  for 
educating  students  that  merely  presents  a  fantasy  world  for  them  to  explore.  Pairs  of 
students  at  different  locations  with  a  teacher  represented  in  the  virtual  space  were 
tested  using  the  environment.  The  study  merely  demonstrated  that  students  could 
navigate  and  use  the  system  but  didn't  test  learning. 
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•These  systems  may  be  able  to  embody  the  information  of  experts'  knowledge 
and  feedback  more  easily  and  have  a  virtual  expert  available  at  all  times. 

•  Improved  distance  learning  can  actually  provide  more  interaction  possibilities. 


Potential  problems 

•  Better  technology  is  needed  for  many  reasons.  First,  the  network  connections 
need  to  be  faster  and  more  reliable  than  they  are  right  now.  In  some  cases,  access  to 
connections  may  also  be  an  issue.  Second,  technology  is  rapidly  changing  and 
therefore  the  infrastructure  must  be  setup  to  adapt  to  these  changes.  Third,  there  is 
often  not  enough  funding  for  upkeep  or  enough  specialized  individuals  to  keep  the 
technology  working. 

•Assessment:  More  assessment  needs  to  be  performed  at  all  levels.  In  order  to 
make  sure  that  both  learners  and  instructors  can  use  the  equipment  (human-computer 
interface  usability).  This  needs  to  go  beyond  satisfaction.  Virtual  learning 
environments  also  need  to  be  examined  for  how  well  learners  acquire  the  skills  and 
knowledge  that  they  need.  Immersive  virtual  learning  environments  need  to  be 
compared  to  learning  the  same  skills  and  knowledge  in  the  real  world  (classes,  field 
training,  etc.),  against  synthetic  task  environments,  and  non-immersive  virtual  reality. 
Finally,  the  systems  need  to  be  evaluated  for  whether  learners  can  transfer  their  skills 
and  knowledge  to  the  real  world. 

•More  research  has  been  performed  on  individuals  using  immersive  learning 
environments.  However,  even  if  learning  and  transfer  is  established  for  individuals 
(either  in  co-located  or  distributed  learning  environments),  findings  for  individuals 
may  not  translate  to  teams.  Therefore  some  additional  research  may  be  needed  for 
team  training  in  distributed  virtual  learning  environments. 

•  In  distributed  training,  asynchronous  connections  between  an  instructor  and 
learner  is  assumed.  However,  with  synchronous,  immersed  learning  environments,  it 
may  be  difficult  to  manage  the  teams  of  learners  and  they  may  expect  the  instructor  to 
be  present. 

•  Cultural  needs  of  learners  who  are  at  a  distance  may  need  to  be  taken  into 
account,  especially  if  they  are  in  different  countries  (Bryant  &  Davis,  2005;  Helmreich  & 
Merritt,  1998). 

•The  cost  may  be  too  high  (especially  for  schools)  for  both  development  and 
appropriate  evaluation.  (Extensive  evaluation,  especially  whether  people  actually 
learn  from  a  training  system,  is  often  not  performed  because  there's  a  push  to  get  a 
system  in  the  field.) 

•Because  many  locations  where  a  distributed  training  system  might  be  used 
have  different  types  of  hardware  (and  sometimes  even  different  software),  cross¬ 
platform  issues  need  to  be  taken  into  account. 

•There's  a  lack  of  standards  in  the  field  (see  also  Shaffer  et  al.,  2004;  Stytz  & 
Banks,  1998). 
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•The  time  to  train  individuals  and  instructors  might  be  high  given  the 
complexity  of  such  systems  (see  Pajo,  2001). 


Other  issues 

One  of  the  problems  with  examining  a  topic  such  as  virtual  learning 
environments  is  that  many  fields  are  trying  to  develop  and  integrate  virtual  reality  into 
their  general  systems  and  education/ training  curriculums.  The  terminology  across 
areas  is  often  slightly  different.  In  this  paper,  we  did  our  best  to  summarize  published 
works  but  some  items  might  have  been  missed  if  they  were  not  using  the  same 
terminology,  and,  as  before,  some  items  appear  in  conference  proceedings  but  have  not 
gone  through  the  rigorous  evaluation  of  journal  acceptance.  Although  this  paper  did 
briefly  cover  medicine  and  education  (though  not  extensively),  the  gaming  arena  is 
constantly  developing  new  and  better  technology  (Hall,  Hong,  Marshall,  Garrity,  & 
Green,  2005)  to  satisfy  customers'  demands.  This  creates  a  new  area  of  research  -  the 
adaptation  of  software  developed  for  commercial  applications  (and  systems)  to 
military  ones.  (See  Bakker  &  Button,  2003,  for  an  example  of  integrating  a  synthetic 
training  environment  with  existing  systems.)  Although  not  insurmountable,  the 
differences  can  still  be  extensive. 


Future  Directions 

The  ideal  system  would  be  a  virtual  environment  in  which  individuals  or  teams 
could  learn  or  practice  what  they  know.  Individuals  or  team  members  would  all  be 
immersed  and  interact  as  if  they  were  in  the  real  world.  The  system  would  have  to 
contain  ways  for  team  members  to  communicate  both  verbally  and  non-verbally.  It 
would  have  to  include  a  background  tutoring  system  that  models  ideal  behavior  and 
can  provide  corrective  feedback.  (For  team  training  this  would  be  quite  difficult 
because  each  team  member  would  have  to  be  monitored,  along  with  the  overall  team 
performance.)  The  system  would  be  able  to  work  in  a  distance  fashion  much  as 
MOOs8.  This  would  allow  teams  to  train  before  they  arrive  to  do  a  job.  Finally,  all  of 
these  aspects  would  be  evaluated  for  their  ability  to  train  people  and  not  just  for  user 
satisfaction.  The  following  paragraphs  describe  different  issues  that  still  need  to  be 
addressed  within  this  vision. 


8  MOOs  are  network  accessible,  multi-user,  programmable,  interactive  systems  well- 
suited  to  the  construction  of  text-based  adventure  games,  conferencing  systems,  and 
other  collaborative  software.  Their  most  common  use,  however,  is  as  multi-participant, 
low-bandwidth  virtual  realities  (Wikpedia,  2006).  Ideally,  we  would  want  high- 
bandwidth  virtual  realities  for  this  situation.  (See  Schalger,  Fusco,  &  Schank,  1998  for  a 
description  of  one  MOO.) 
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The  most  important  aspect  of  any  virtual  environment  system  is  the  ability  to 
transfer  the  knowledge  learned  to  the  performance  in  the  real  world.  Virtual  learning 
environments  have  had  many  benefits  for  learners  (Rose  et  al.,  2000),  however,  only 
transfer  of  sub-skills  has  been  demonstrated  but  overall  transfer  to  real-world 
environments  has  not  been  validated  (Brooks,  Attree,  Rose,  Clifford,  &  Leadbetter, 

1999;  Cohn,  Helmick,  Meyers  &  Burns,  2000;  Witmer,  Bailey,  &  Knerr,  1996;  Witmer  & 
Sadowski,  1998). 

What  is  needed  in  the  intersection  of  virtual  learning  environments  and  team 
training?  Improvements  in  simulations  and  hardware  are  needed  for  true  virtual 
learning  environments.  It's  not  that  training  systems  don't  exist  but  that  they  could  be 
vastly  improved  (see  results  from  Tromp  et  al.,  2003).  The  issues  are  different  if  the 
training  system  is  supposed  to  supply  the  other  team  members  (virtual  humans;  see 
Garau  et  al.,  2005)  or  whether  all  the  real  human  team  members  will  be  represented  in 
the  virtual  environment  for  training.  This  last  possibility  would  allow  for  distance 
team  training  in  a  virtual  environment.  In  both  cases,  though,  the  technology  has  more 
work  to  do  before  a  virtual  human  will  be  viewed  in  the  same  way  as  a  real  human. 

A  true  training  system  would  constantly  monitor  the  learner  (or  learners)  and 
provide  feedback.  For  example,  feedback  would  necessitate  a  way  to  analyze  the 
team's  performance  as  it  is  happening.  Both  the  actions  of  the  team  need  to  be 
analyzed  as  they  occur  and  the  communication  of  the  team  members.  New  technology 
such  as  those  proposed  by  Foltz  (Foltz,  et  al.,  2003;  Foltz  &  Martin,  2004;  Foltz,  et  al., 
2006)  needs  to  be  integrated  into  existing  technologies. 

Aside  from  system  improvement,  the  integration  of  learning  technologies  from 
developers  to  adapters  is  often  difficult.  Distance  training  can  be  even  more  difficult 
because  systems  at  different  sites  need  to  be  coordinated.  Further  instructors  need  to 
be  trained  both  at  location  with  current  technology  and  across  sites  to  coordinate 
curriculum  and  training  issues. 

Research  is  needed  to  improve  both  virtual  environments  generally  and  virtual 
learning  environments  in  particular,  in  order  to  both  create  and  demonstrate  transfer  of 
training.  There  is  no  doubt  that  government  sponsored  research  and  technology 
corporations  will  continue  to  develop  better  hardware  and  software  for  the  task  (see 
Hall  et  al.,  2005,  for  commentary);  however,  aside  from  research  that  merely  creates 
new  and  better  virtual  environments,  virtual  learning  environments,  focus  on 
evaluation  (whether  something  was  learned  and  not  merely  "liked")  is  needed  in  order 
to  demonstrate  that  these  systems  really  produce  learning. 
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